Imaging model and apparatus

ABSTRACT

A system and method for making an imaging model and an imaging apparatus. An apparatus for image processing may include an optical element having a field of view, and image sectoring element coupled to the optical element, the image sectoring element configured to sector the field of view in a plurality of areas, and an image processor coupled to the image sectoring element, the image processor configured to process an image in accordance with the plurality of areas. Methods to make the foregoing apparatus are also described.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.60/875,649; filed Dec. 19, 2006, titled “A Model of Wide-Angle Foveationfor All-purpose use.”

INCORPORATION BY REFERENCE

References cited within this application, including patents, publishedpatent applications other publications, and the U.S. ProvisionalApplication No. 60/875,649; filed Dec. 19, 2006, are hereby incorporatedby reference in their entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

THE NAMES OF THE PARTIES TO A JOINT RESEARCH AGREEMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND

1. Field

This disclosure is generally related to an imaging model and anapparatus and in particular to a foveation-related imaging model and asystem based on foveation-related imaging.

2. Description of Related Art

A typical human retina-like image sensor, that is, a fovea visionsensor, is applicable to several uses. Such a space-variant image sensorrealizes facilitates observing a wider field-of-view (FOV) having a muchsmaller number of data, and observing the central FOV in more detailthan other parts of the FOV. Log-polar (LP) mapping is used as a typicalmodel for image representation. This mapping is inspired by analyticformulation from biological observation of the primate visual system.This mapping has been applied to computer vision computationally toproduce an LP vision chip having CCD or CMOS technologies. The LPmapping is effective not only for a significant image data reduction, asthe human retina does, but also suitable for generating geometricalrotation and scale-invariant feature because of mathematical property ofLP mapping.

Another method to acquire the log-polar image, being an optical approachalso exists. This approach usually combines the specially-made WideAngle Foveated (WAF) lens with a commercially available conventionalCartesian vision chip, where photosensitive element is size-invariant,although the LP chip approach combines the specially-made chip withlogarithmic size-variant photosensitive elements having a conventionallens. The optical approach can realize a more complicated combination ofdifferent coordinate systems more easily than the specially-made chip.The WAF lens can provide higher resolution in the central FOV because ofits optical magnification factor (M.F.).

A camera's view direction control is quite essential for the foveavision system suggesting to take account of overt-attention, that is, atype of attention when the camera is dynamically moved. Another type iscovert-attention, that is, attention when the camera is staticallyfixed. A rotation, scale, and translation-invariant property isapplicable for pattern recognition. Fourier-Mellin Transform is known asan algorithm to generate such a property. However, generally, aCartesian image is not rotation- and scale-invariant, and an LP image isnot translation-invariant, that is, translation causes geometricaldeformation of projection in the LP coordinates. An overt-vision systemwith the fovea vision sensor can combine such two types of image for areliable pattern recognition, because precise camera view directioncontrol to a target, using the Cartesian image, reduces the distortionin the LP image. In addition, if the FOV is represented by a sphericalprojection, it is useful for the camera's view direction control.

An LP image acquired by the space-variant fovea sensors, is transformedinto a system of Cartesian coordinates, generating a Cartesian image,because Fourier-Mellin Transform needs Cartesian coordinates in order toextract rotation, scale and translation-invariant features. However,this does not mean that the Cartesian image is more suitable as arepresentation of an original input image, because the Cartesian imageremapped conversely from the space-variant input image has a higherresolution in its central FOV than that in the opposite case, i.e., fromthe Cartesian image to the LP image.

A WAF lens input image is shown in FIG. 1 comparing with the pinholecamera (PHC) lens. This PHC image has the same FOV, that is, the sameview angle and the same number of pixels. The WAF lens has about 120degrees wide FOV and adequate high resolution in the central FOV.

FIG. 2 illustrates plots related to WAF, LP, FE, and PHC lenses. FIG. 2(a) shows object height h vs. image height r. FIG. 2( b) shows objectheight h vs. magnification factor (M.F.) dr/dh. FIG. 2( c) shows objectheight h vs. M.F. r/h.

FIGS. 3( a-e) illustrate an original image, and images from a simulationof a WAF lens, an LP lens, an FE lens, and a PHC lens in that order.

FIG. 4 illustrates various plots for prior-art FE, PHC, and Kuniyoshilenses. FIG. 4( a) shows object height h vs. image height r. FIG. 4( b)shows object height h vs. M.F. dr/dh. FIG. 4( c) shows object height hvs. M.F. r/h.

FIGS. 5( a-c) show exemplary test images under three scalings of 1.0,0.75, and 1.5. FIGS. 5( d) and (f) are the LP lens image, and aKuniyoshi lens (K lens) image simulated by the distribution of M.F. ofthe actual K lens. Both are extracted from FIG. 5( a) in conditions ofθ_(max)=π/3, h_(max)=1, h₀=0.026, and h₁=0.21.

FIG. 6 shows LP images from an LP lens (left) and a K lens (right) underscaling of 0.75, 1.0, and 1.5 when θ_(max)=π/3, h_(max)=1, h₀=0.026, andh₁=0.21.

FIG. 7 shows plots of object height h vs. length on image plane for LPand K lenses illustrating an accuracy of scale-invariance by prior-artlenses such as LP lens, and K lens. A broken line, and a bold solid lineshow the LP lens, and the K lens, respectively.

Vision sensors such as a CCD camera can acquire more information thanother sensors. Further, wide-angle is more convenient tomulti-functional use of visual information to make it possible thatmobile objects, e.g., automobiles and mobile robots, perform flexiblyunder various environments. Typical industrial applications are limitedto single-functional use by a conventional narrow-angle vision sensor,e.g., an inspection system, and a medical application. Generally, thereis a trade-off between a wide-angle and a high resolution. A wide-angleand a high resolution at the same time normally causes an enormousincrement of the number of pixels per an image posing a serious problemfor data transmitting and real-time processing.

It may be helpful to use a foveated visual information based on humanvisual property. A human eye has a 120 degree wide-angle visual field.The visual acuity is more near a fovea in the central area of retina andbecomes lower towards a peripheral area. Methods to reconstruct afoveated image based on log-polar mapping by a computer, and to obtainit using a space variant scan CCD exist. A camera's view direction iscontrolled to acquire target information in detail at attention point.This system is called Wide Angle Foveated Vision Sensor (WAFVS) system.

The WAFVS system is composed of image input part, view direction control(VDC) part, and image processing part. Image input part has 2 CCDcameras with a special super wide-angle lens and image capture device.The special lens, named WAF lens (FIG. 8), plays a major role in thispart. This lens is attached to a commercially available CCD camera andoptically realizes a WAF image with 120 degrees wide visual field andlocal high resolution in its central area at the same time. Stereovision with such input image provides 3D information with adequateaccuracy, and further wider view area at the same time. FIG. 9 showscharacteristics of left camera's image height r_(L) on CCD image planeversus incident angle θ_(L) to WAF lens. For a comparison, image heightr_(per) ^(L) of the PHC lens with the same visual field and the sameamount of information are also shown. FIG. 1 shows the images by a WAFlens and a PHC. These curved lines are represented as Equ. (1) (WAFlens) and Equ. (2) (PHC lens). The inclination of each curve shows imageresolution along a radial direction of the visual field. A typical WAFlens has a higher resolution in its central area and, on the other hand,has lower resolution in the peripheral area, compared to that of the PHClens.

$\begin{matrix}{r_{i} = {{f_{0}^{i}\theta_{i}^{3}} + {f_{1}^{i}\theta_{i}^{2}} + {f_{2}^{i}{\theta_{i}\mspace{14mu}\lbrack{pixels}\rbrack}}}} & (1) \\{{r_{per}^{i} = {r_{\max}^{i}{\frac{\tan \; \theta_{i}}{\sqrt{3}}\mspace{14mu}\lbrack{pixels}\rbrack}}},} & (2)\end{matrix}$

where each f_(k) ^(i) (k=0,1,2) is a coefficient determined by cameracalibration, and r_(max) ^(i) is image height of 60 degrees by Equ. (1).Subscript i means left camera or right camera.

Input image from WAF lens is suitable for multi-purpose andmulti-functional use, because it has two different kinds ofcharacteristic, i.e., wide-angle and local high resolution. White andblack circles in each of FIG. 1( a) and (b) show incident angles with 10and 30 degrees. The peripheral area of WAF image 30 to 60 degrees isabout 40% of the whole visual field, compared to that of PHC lens imagewith about 90%. This area with less pixels facilitates peripheralvision, e.g., detecting an intruder and localization, and so on. On theother hand, the central area (0 to 10 degrees) of WAF image is about 10%compared to that of pinhole camera image with about 1%. This area hasadequate high resolution for central vision, e.g., recognizing objectsbased on color, shape and pattern, acquiring more accurate 3Dinformation and so on. The intermediate area (10 to 30 degrees) is forboth central and peripheral visions.

Camera's view direction control (VDC) to change an attention point inview area rapidly using camera mover is effective for a WAF image. VDCpart has VDC mechanism and four stepping motors. These motors realizeneck pan rotation and left and right two cameras' tilt and vergence bytwo kinds of eye movement such as human saccade (rapidly) and pursuit(precisely).

Image processing part is composed of multiple computers havingmulti-task operating system (OS) under wireless and wired LAN. This partis characterized by flexible parallel image processing function based ontimely task distributing (FIG. 10). This function has been investigatedto carry out various image processing tasks in a parallel andcooperative manner. Several kinds of image processing with variouslevels are executed in parallel, in a distributed manner or in aselective manner, based on each processor's load. The main computerplays a role of a file server and has shared information among thecomputers and among the multiple tasks. Combination with camera's VDCextends application of the WAFVS system. Instead of mobile robotnavigation, it seems to be effective for multi-functional applicationsuch as object tracking and simultaneous recognition.

A rational mobile robot navigation based on multi-functional use of WAFimage exists as shown in FIG. 11( a). The navigation is based on twotasks of central and peripheral visions to utilize the WAF lensproperty. Central vision plays a role to plan an obstacle avoidancecourse from more accurate 3D information. On the other hand, peripheralvision plays a role to revise the locational information under odometry.The planned course and locational information are shared between thetasks and are updated periodically to improve quality of the navigationcooperatively. For example, the revised locational information improvesthe planned course, as regards a target point on the planned course, andobjective moving distance and rotating angle are calculated. Thecalculated values are input to a computer for driving control of mobilerobot. FIG. 11( b) shows a flow chart of this navigation. The period ofperipheral vision is much shorter than that of central vision. Thisset-up is based on a model similar to human visual informationprocessing, because peripheral vision is connected to mobile robotcontrol more closely.

FIG. 12( a) shows the visual point coordinate systemO_(i)-X_(i)Y_(i)Z_(i) where the camera's optical axis coincides at Y_(i)axis and the origin is the visual point of WAF lens (i=L, R).Coordinates (u_(i)′, v_(i)′) on an input image are corrected to visualfield coordinates (u_(i), v_(i)) by dot aspect ratio K and correspond to(θ_(i), φ_(i)), incident direction to the visual point, where (I_(u)^(i), I_(v) ^(i)) is the image center. (u_(per) ^(i), v_(per) ^(i)) areperspective coordinates transformed from visual point coordinates usingEqu. (2).

FIG. 12( b) shows left camera's visual point coordinate systemO_(L)-X_(L)Y_(L)Z_(L), robot coordinate system O_(c)-X_(c)Y_(c)Z_(c) andworld coordinate system O_(w)-X_(w)Y_(w)Z_(w). It is assumed that roadplane is ideally flat. In FIG. 12( b), ψ_(1c) and ψ_(2c) are deflectionangles of binocular camera mover to the robot in pan and tilt directionsrespectively. The origin, O_(c), of robot coordinates is the robotcenter and the world coordinates are (X_(robo), Y_(robo), 0). P is alength from neck rotation axis to the robot center, H is height ofcamera's visual point from road plane and B is base line length betweenleft and right camera's visual points, α is an angle between the Y_(c)axis and the Y_(w) axis.

Obstacle avoidance course is determined using 3D information obtained bypassive parallel stereo method as a task of central vision.Determination of avoidance course is based on road map. Road map has twodimensional X_(w) and Y_(w) axes of world coordinate system, and hasenvironmental information such as walls, road boundary lines, detectedobstacles and so on. This road map is different from that often used inpath planning research such as a Voronoi graph. FIG. 13( i) shows theway to presume an area where obstacles exist. Here a camera's viewdirection is assumed to be parallel to Y_(w) axis. The steps involvedare:

(a): Road plane is divided to small square blocks with a side of 5 [cm],(b), (c): Each 3D measured point (x, y, z) is voted to the correspondingblocks considering measurement errors caused by CCD elementdigitization. Δy and Δx, errors in the directions of Y_(w) (Y_(L)) axisand X_(w) (X_(L)) axis respectively are calculated using Equs. (3) and(4).

$\begin{matrix}{{\Delta \; y} = {{{\frac{\partial y}{\partial\theta_{L}}} \cdot {\Delta\theta}_{L}} + {{\frac{\partial y}{\partial\theta_{R}}} \cdot {\Delta\theta}_{R}} + {{\frac{\partial y}{\partial\varphi_{L}}} \cdot {\Delta\varphi}_{L}} + {{\frac{\partial y}{\partial\varphi_{R}}} \cdot {\Delta\varphi}_{R}}}} & (3) \\{{{\Delta \; x} = {{{\frac{\partial x}{\partial\theta_{L}}} \cdot {\Delta\theta}_{L}} + {{\frac{\partial x}{\partial\theta_{R}}} \cdot {\Delta\theta}_{R}} + {{\frac{\partial x}{\partial\varphi_{L}}} \cdot {\Delta\varphi}_{L}} + {{\frac{\partial x}{\partial\varphi_{R}}} \cdot {\Delta\varphi}_{R}}}},} & (4)\end{matrix}$

where Δθ_(i) and Δφ_(i) (i=L,R) are errors of incident angle in theradial and tangential directions of visual field respectively, caused byCCD element digitization. FIG. 13( ii) shows the above errors,and (d): The obstacle is presumed to exist in highlighted blocks basedon a threshold.

FIG. 14( i) shows a flow to determine avoidance course on road map. Thehatched area in each step shows that there are no obstacles. Obstacleinformation is given to road map with offset to avoid collision. Whitecircles are data points on the determined avoidance course. Informationof road boundary lines is acquired by algorithm of peripheral visiondescribed in the next subsection.

FIG. 14( ii) shows contour graphs of depth error Δy of each point on theplane including 2 camera's view lines. Here base line length B is 300[mm]. The error is calculated based on Equs. (3) and (4) by computersimulation. Each value is represented as a ratio Δy/y. For a comparison,depth error by PHC lens image represented by Equ. (2), is shown. Brokenlines are boundary lines of camera's view field, and the area insidethem has binocular information. As shown in FIG. 14( ii), depth may bemeasured with higher accuracy in small incident angles to left camera bya WAF lens than by a PHC lens. PHC lens cannot measure depth within 2%error only inside the near range of about 0.6 m ahead. On the otherhand, WAF lens can measure depth with the similar accuracy in thefarther range of about 3.2 m ahead.

A method exists to obtain location and orientation from a single CCDcamera using two road boundary lines projected in the peripheral area,as Two Parallel Line (TPL) algorithm described in FIGS. 15( a), 15(b),and 16. This method realizes to detect locational information with ahigher accuracy, because the peripheral area has a higher resolution intangential direction while having fewer pixels. Because a length ofblack circle with 30 degree in WAF image (FIG. 1( a)) is longer thanthat of FIG. 1( b). It is assumed that there are two parallel boundarylines (l_(A) and l_(B)) on flat road plane and there is no rotationabout the optical axis of the camera. In addition, road width Wandheight H of visual point from the road plane, are assumed to be known.

The left camera's visual point O_(L) (X_(o) ^(L),Z_(o) ^(L)) iscalculated from two planes including each boundary line from coordinatesystem O_(L)-X_(L)Y_(L)Z_(L) and a related coordinate systemO_(L)-X_(L)′Y_(L)′Z_(L)′ which has the same origin as shown in FIG. 15(b). As to the O_(L)-X_(L)′Y_(L)′Z_(L)′, Y_(L)′ axis is parallel to twoboundary lines and X_(L)′ axis is horizontal to road plane. These planesare represented as Equ. (5).

$\begin{matrix}{Z^{\prime} = {{- a_{i}}X^{\prime}\mspace{14mu} \left( {{i = A},B} \right)}} & (5) \\{{O_{L}\left( {X_{o}^{L},Z_{o}^{L}} \right)}\mspace{14mu} {is}\mspace{14mu} {calculating}\mspace{14mu} {using}\mspace{14mu} {a_{i}.\left\{ \begin{matrix}{X_{o}^{L} = {\frac{{Wa}_{A}}{a_{A} - a_{B}} - {W/2}}} \\{Z_{o}^{L} = \frac{{Wa}_{A}a_{B}}{a_{A} - a_{B}}}\end{matrix} \right.}} & (6)\end{matrix}$

Road width W is calculated from Equ. (7), because visual point heightH(=Z_(o) ^(L)) is known. This means that it is possible to navigate amobile robot in an unknown corridor environment as well.

W=H/a _(A) −H/a _(B)  (7)

Camera's view direction relative to boundary lines, represented by panangle ψ₁ and tilt angle ψ₂, is calculated from the vanishing point inperspective coordinate system. Orientation α of a mobile robot in worldcoordinate system is represented as Equ. (8), when camera mover isdeflected with pan angle ψ_(1c) from robot coordinate system.

α=ψ₁−ψ_(1c)  (8)

FIG. 16 shows a flowchart of the TPL algorithm using Hough transform.This algorithm detects locational information rapidly, when parts ofboundary lines are invisible by obstruction or lighting conditions.

If camera mover is fixed at the mobile robot, accuracy of the locationaland 3D information ahead the road is reduced, as orientation of themobile robot gets larger. Camera's view direction control solves thisproblem by keeping a view direction parallel to road boundary lines.Rotating angle Δψ₁ is calculated from Equ. (9).

Δψ₁=−ψ₁−({circumflex over (α)}₂−{circumflex over (α)}₁)  (9),

where {circumflex over (α)}₁ is an estimated value of mobile robotorientation just after image input, based on odometry, and {circumflexover (α)}₂ is an estimated value just after locational information iscalculated from the peripheral area.

Locational information of the mobile robot is revised periodically withα−({circumflex over (α)}₂−{circumflex over (α)}₁) andX_(robo)−({circumflex over (X)}_(robo2)−{circumflex over (X)}_(robo1)),where {circumflex over (X)}_(roboi) is a value of mobile robot locationby odometry just after image input (i=1) and just after locationalinformation is calculated (i=2). If a road width calculated from Equ.(7) is much different with the known width W, locational information isnot revised.

When using the TPL algorithm, the optimal value of height h of camera'svisual point to detect camera's horizontal position w accurately exists.The relation between w and h is examined by computer simulation, whencamera's view direction is parallel to road boundary lines. FIG. 17shows two projected road boundary lines (represented by φ_(n) (n=A,B))in visual field coordinate system. Measured error, Δw, of horizontalposition caused by CCD digitization errors, Δφ_(A) and Δφ_(B), iscalculated from Equs. (10) and (12).

$\begin{matrix}\left\{ {\begin{matrix}{{\Delta\varphi}_{n} = \frac{1}{r{{\cos \; \varphi_{n}}}}} & {\left( {{{- \pi} \leq \varphi_{n} \leq {{- \frac{3}{4}}\pi}},{{{- \frac{1}{4}}\pi} \leq \varphi_{n} \leq {\frac{1}{4}\pi}},{{\frac{3}{4}\pi} \leq \varphi_{n} \leq \pi}} \right),} \\\frac{1}{r{{\sin \; \varphi_{n}}}} & {\left( {{{{- \frac{3}{4}}\pi} \leq \varphi_{n} \leq {{- \frac{1}{4}}\pi}},{{\frac{1}{4}\pi} \leq \varphi_{n} \leq \frac{3}{4}}} \right),}\end{matrix}{where}\mspace{14mu} n\mspace{14mu} {is}\mspace{14mu} A\mspace{14mu} {or}\mspace{14mu} {B.}} \right. & (10) \\{w = \frac{W\mspace{11mu} \tan \mspace{11mu} \varphi_{A}}{{\tan \mspace{11mu} \varphi_{A}} - {\tan \mspace{11mu} \varphi_{B}}}} & (11) \\{{\Delta \; w} = {{{\frac{\partial w}{\partial\varphi_{A}}}{\Delta\varphi}_{1}} + {{\frac{\partial w}{\partial\varphi_{B}}}{\Delta\varphi}_{B}}}} & (12)\end{matrix}$

FIG. 18 shows contour graphs of error Δw in each position (w/W, h/W).FIG. 18( a) is from the WAF lens and FIG. 18( b) is from the PHC lens.Each value of Δw is represented with percentage of road width W. Δw iscalculated from Δφ_(A) and Δφ_(B) on a circle with a radius of 0.875 andthose on a circle with a radius of 0.405 as to WAF lens and PHC lens,respectively. These radii correspond to about 35 degree incident angle.There is the height of visual point to make Δw minimum as to eachhorizontal position w (shown as a broken line in FIG. 18). The optimalheight gets smaller, as w gets closer to zero or 1. The error becomesmore sensitive to change of was h gets closer to zero and w gets closerto zero or 1. It is noted that the WAF lens can measure w with a higheraccuracy than the PHC lens, because the resolution is higher intangential direction of the same incident angle.

A mobile robot used for an experiment is shown in FIG. 19. It is a FrontWheel System vehicle with an active front wheel, for both driving andsteering, and two rear wheels. On the robot, there are two computers forimage processing and driving control of mobile robot, which run eachprocess in parallel and share locational information of the mobilerobot, by communication. Locational information is estimated fromrotations measured by rotary encoders for driving and steering, and isrevised by values detected from peripheral vision to improve quality ofthe planned course and navigation.

As shown in FIGS. 20( i)(a) and (b), two kinds of navigation experimentare carried out, where white boards (width 20 [cm]×height 55 [cm]) areplaced as obstacles on a carpet (width 137.5 [cm]). Tilt angle ψ₂ ofleft camera's view direction, is 0 degrees, and height H of the camera'svisual point is 64.5 [cm], and horizontal distance P between the robotcenter and neck rotation axis of camera mover is 53 [cm]. Collisionoffset between obstacles and the mobile robot is set with 20 [cm]. Themobile robot is set to move with velocity of about 10 [cm/s].

FIG. 20( ii) shows results of the experiment. White triangles are targetpoints on the planned courses, and black dots are points by (X_(robo),Ŷ_(robo)) on actual courses, where X_(robo) is a measured value by theTPL algorithm and Ŷ_(robo) is an estimated value by odometry. Crossesare obstacles. They are plotted with world coordinates respectively. Inthese experiments, steering is controlled to follow trajectories basedon cubic function fitted to target points. It seems that the gap betweentwo courses is caused by delay of the steering control. A target pointclose to 300 mm in FIG. 20( ii)(b) is influenced by errors of locationalinformation from the TPL algorithm.

A fovea sensor gives a foveated image having a resolution that is higherin the central FOV and decreases rapidly as going from a central area tothe periphery. That is, the resolution of the fovea sensor isspace-variant. Thus, the fovea sensor functions by wide-angle FOV and indetail in the central FOV using largely-reduced number of data.Log-polar mapping is often used for a model of the foveated image. Thelog-polar mapping is inspired by analytic formulation from biologicalobservation of the primate visual system. Log-polar is applied this tocomputer vision computationally and to produce a log-polar vision chipwith CCD or CMOS technologies. The log-polar mapping is not onlyeffective for a drastic reduction in image data, as the human retinadoes, but is also suitable for generating geometrical rotation andscale-invariant feature easily.

Another method and a wide-angle lens exist to acquire the foveatedimage. Such a wide-angle lens combines a specially-made Wide AngleFoveated (WAF) lens with a commercially available conventional Cartesianlinear-coordinate vision chip, where photosensitive elements arearranged uniformly. On the other hand, the former approach combines aconventional lens with the log-polar chip, where the size ofphotosensitive element is uniform in the fovea and changeslogarithmically in periphery.

A special wide-angle lens is known as a model that combines planarprojection and spherical projection. This lens achieves foveation bydistorting a part of spherical projection using a logarithmic curve inorder to bridge ‘linear’ planar projection and ‘linear’ sphericalprojection. This part of the FOV, that is, spherical logarithmic part,has rotation- and scale-invariant (RS-invariant) property.

BRIEF SUMMARY

Embodiments of the present disclosure provide a system and method formaking an imaging model and an imaging apparatus. The present disclosureteaches how to make an imaging model and a system.

Briefly described, in architecture, one embodiment of the system, amongothers, can be implemented as follows.

An apparatus for image processing may include an optical element havinga field of view, an image sectoring element coupled to the opticalelement, the image sectoring element configured to sector the field ofview in a plurality of areas, and an image processor coupled to theimage sectoring element, the image processor configured to process animage in accordance with the plurality of areas.

The present disclosure can also be viewed as providing a method ofmodeling an image. The method may include providing an optical element,assigning a field of view of the optical element, sectoring the field ofview in a plurality of areas, and processing an image in accordance withthe plurality of sectored areas.

Other systems, methods, features, and advantages of the presentinvention will be, or will become apparent, to a person having ordinaryskill in the art upon examination of the following drawings and detaileddescription. It is intended that all such additional systems, methods,features, and advantages included within this description, be within thescope of the present disclosure, and be protected by the accompanyingclaims.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. Components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating principles of the present invention. Moreover, in thedrawing, like-referenced numerals designate corresponding partsthroughout the several views.

FIG. 1 shows images of WAF lens and pinhole camera (PHC) lens.

FIG. 2 shows a comparison of plots of prior-art lenses, in terms ofobject height.

FIG. 3 shows a simulation of WAF lens, LP lens, lens, PHC lens(Prior-art) images.

FIG. 4 shows a comparison of plots of prior-art lenses.

FIG. 5 shows test images with different scale, for prior art lenses suchas LP lens image, and K lens image by distribution of M.F. of the actualK lens.

FIG. 6 shows LP images from prior art lenses such as the LP lens (left)and K lens image (right).

FIG. 7 shows an accuracy of scale-invariance by prior-art lenses such asLP lens, and K lens.

FIG. 8 shows a prior-art compound system and a picture of WAF lens.

FIG. 9 shows an image height-incident angle plot of input images, shownin FIG. 1, of a prior-art WAF lens and pinhole camera.

FIG. 10 shows a scheme of a prior-art WAFVS system and timely taskdistributing.

FIG. 11( a) shows navigation based on multi-functional use of aprior-art WAFVS.

FIG. 11( b) shows a flow chart of cooperative navigation between aprior-art central vision and a prior-art peripheral vision.

FIG. 12( a) shows visual field coordinates and visual point coordinatesin prior art.

FIG. 12( b) shows Left camera's visual point coordinates, robotcoordinates and world coordinates in prior art.

FIG. 13( i) shows a way to presume an area where obstacles exist.

FIG. 13( ii) CCD digitization errors and depth error Δy and width errorΔx of 3D measurement.

FIG. 14( i) shows a flow to plan obstacles avoidance course.

FIG. 14( ii) shows contour lines of depth error.

FIG. 15( a) shows parameters of robot's and camera mover's location andorientation.

FIG. 15( b) shows a sketch of coordinate systemO_(L)-X_(L)′Y_(L)′Z_(L)′.

FIG. 16 shows a flowchart of TPL algorithm.

FIG. 17 shows two boundary lines in visual field when camera's viewdirection is parallel to these lines.

FIG. 18 shows contour graphs of horizontal position error Δw.

FIG. 19 shows a prior-art WAFVS system and a mobile robot.

FIG. 20( i) shows experimental environments for obstacle avoidancenavigation.

FIG. 20( ii) shows experimental results of prior-art obstacle avoidancenavigation.

FIG. 21 shows an embodiment of a camera model of the present disclosurebased on combination of planer projection and spherical projection.

FIG. 22 shows a plots for an AdWAF image in terms of object height.

FIG. 23 shows an original image and a simulation of an AdWAF image.

FIG. 24 shows an AdWAF image extracted from an actual image by WAF lens.

FIG. 25 shows plots for the AdWAF image.

FIG. 26 shows test images with different scales, and an AdWAF image.

FIG. 27 shows LP images from the LP lens (left) and AdWAF image (right).

FIG. 28 shows plots for K lens (AdWAF) and AdWAF image.

FIG. 29 shows another embodiment of a camera model of the presentdisclosure based on a combination of planar projection and sphericalprojection.

FIG. 30 shows a comparison of AdWAF image and linear-coordinate image.

FIG. 31 shows plots for an AdWAF image.

FIG. 32 shows a target image and simulated images of the another AdWAFmodel.

FIG. 33 shows an AdWAF image extracted from an actual image.

FIG. 34A shows foveation models of LP lens log-polar chip, for example.

FIG. 34B shows plots for an AdWAF image in terms of object height.

FIG. 35( i) shows an AdWAF image, an LP lens image and a K lens image.

FIG. 35( ii) shows three target images with different scales (α=0.75, 1and 1.5).

FIG. 35( iii) shows polar images of AdWAF image, LP lens image and Klens image when r_(max)=64, θ_(max)=π/3, θ₀=2.584 [°], θ₁=20.0 [°] andθ₂=34.715 [°].

FIG. 36 shows a scale-invariant property of an AdWAF image, an LP lens,and a K lens.

FIG. 37 shows an influence of skew by an AdWAF image and LP lens.

FIG. 38 shows Polar images of AdWAF image and LP lens model when ψ=0, 5and 10 [°] from the left.

FIG. 39 illustrates a flowchart of a method of the present disclosure.

FIG. 40 illustrates a block diagram of an exemplary embodiment of anapparatus of the present disclosure.

FIG. 41A shows a 3D test pattern.

FIG. 41B shows 3D test pattern image by an AdWAF lens.

FIGS. 42A and 42B show detecting a vanishing point from two roadboundary lines

FIG. 43 shows a face detection using color information in the centralFOV.

FIG. 44 shows a moving object detection using gray-scale information inthe peripheral FOV as an example of all-purpose use of the AdWAF lens.

DETAILED DESCRIPTION

The present disclosure relates to a system and method for making animaging model and an imaging apparatus.

As a person having an ordinary skill in the art would appreciate, anarrow entering a block or a symbol indicates an input and an arrowleaving a block or a symbol indicates an output. Similarly, connectionsdescribed below may be of any electromagnetic type, such as electrical,optical, radio-frequency, and magnetic.

I. An Embodiment of the AdWAF Model A. Modeling

A relatively correct LP image from the input image by the WAF lens isextracted based on camera calibration. FIG. 21 shows a geometricalsketch of the an embodiment of a camera model, which is based on acombination of planar projection (PP), that is, a perspective projectionby tangent of incident angle, θ, to the lens optical center, andspherical projection (SP), that is, linear to θ. The projection height,p, of this camera model is defined as follows:

if the section of θ is 0≦θ≦θ₁, p=f₁ tan θ,  (13)

else if θ₁≦θ≦θ_(max) (=π/3 [rad]),

p=k·f ₂(θ−θ₁)+f ₁ tan θ₁,  (14)

where f₁ and f₂ are focal lengths to the projection plane and the SPsurface, respectively, and k is a correction factor for continuity ofboth projections,

$\begin{matrix}{k = {\frac{f_{1}}{f_{2}\cos^{2}\theta_{1}}.}} & (15)\end{matrix}$

A model of the WAF image, namely Advanced WAF (AdWAF) imaging, isdefined by the following equations, combining both PP by Equ. 13 and SPby Equ. 14 with both Cartesian and logarithmic coordinates:

$\begin{matrix}{{{{if}\mspace{14mu} 0} \leq \theta \leq \theta_{0}},{r = {r_{\max}c_{1}f_{1}\tan \; \theta}},} & (16) \\{{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{0}} \leq \theta \leq \theta_{1}},{r = {r_{\max}\left( {{c_{2}\log_{a}c_{1}f_{1}\tan \; \theta} + d_{1}} \right)}},} & (17) \\{{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{1}} \leq \theta \leq \theta_{2}},{r = {r_{\max}\left\{ {{c_{3}{\log_{a}\left( {\frac{k \cdot {f_{2}\left( {\theta - \theta_{1}} \right)}}{c_{1}f_{1}\tan \; \theta_{1}} + 1} \right)}} + d_{2}} \right\}}},} & (18) \\{{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{2}} \leq \theta \leq \theta_{\max}},{r = {r_{\max}\left\{ {{c_{4}{f_{2}\left( {\theta - \theta_{2}} \right)}} + d_{3}} \right\}}},} & (19)\end{matrix}$

where r is the image height corresponding to θ, r_(max) is the maximumheight when θ=θ_(max), and c_(i) (i=1,2,3,4) is a scale modificationfactor for adjusting the height of partial images extracted from eachsection of θ. Further, d_(i) (i=1,2,3) is

$\begin{matrix}{{d_{1} = {{c_{1}f_{1}\tan \; \theta_{0}} - {c_{2}{\log_{a}\left( {c_{1}f_{1}\tan \; \theta_{0}} \right)}}}},} & (20) \\{{d_{2} = {{c_{2}{\log_{a}\left( {c_{1}f_{1}\tan \; \theta_{1}} \right)}} + d_{1}}},} & (21) \\{{d_{3} = {{c_{3}{\log_{a}\left( {\frac{k \cdot {f_{2}\left( {\theta_{2} - \theta_{1}} \right)}}{c_{1}f_{1}\tan \; \theta_{1}} + 1} \right)}} + {d_{2}.{Here}}}},{{{if}\mspace{14mu} c_{1}} = {c_{2} = {c_{3} = c_{4}}}},{{{and}\mspace{14mu} r} = {{1\mspace{14mu} {when}\mspace{14mu} \theta_{\max}} = {\pi/3}}},{then}} & (22) \\{k = {\frac{\theta_{2} - \theta_{1} + {\cos \; \theta_{1}\tan \; \theta_{1}}}{\cos^{2}\theta_{1}\tan \; \theta_{0}}.}} & (23) \\{{f_{1} = {bL}},} & (24) \\{{f_{2} = \frac{f_{1}\tan \; \theta_{0}}{\theta_{2} - \theta_{1} + {\cos^{2}\theta_{1}\tan \; \theta_{1}}}},} & (25) \\{{a = {\exp \left( \frac{1}{f_{1}\tan \; \theta_{0}} \right)}},} & (26)\end{matrix}$

where L is the object distance, that is, a distance from the opticalcenter to the object plane, and b is given by

$\begin{matrix}{b = {\frac{1}{L\; \tan \; \theta_{0}}/{\left( {\frac{{\pi/3} - \theta_{2}}{\theta_{2} - \theta_{1} + {\cos^{2}\theta_{1}\tan \; \theta_{1}}} + {\log \frac{\theta_{2} - \theta_{1} + {\cos^{2}\theta_{1}\tan \; \theta_{1}}}{\cos^{2}\theta_{1}\tan \; \theta_{0}}} + 1} \right).}}} & (27)\end{matrix}$

In this case, not only (16)-(19) are continuous but also theirderivatives are continuous.

FIG. 22 shows the image height r, the M.F. dr/dh and r/h in the radialand tangential directions as pertaining to the AdWAF image,respectively, in terms of the object height h, with those of other typesof lens. The h_(max) and r_(max) are normalized to 1 (when θ_(max)=π/3)to compare every type easily. The boundaries of FOV, h₀, h₁ and h₂ are0.1 (=9.826 [°]), 0.4 (=19.107 [°]), and 0.6 (=34.715 [°]),respectively, in case of FIG. 22. Other types of lens, that is, the LPlens, Fish eye (FE) lens, the PHC lens and the WAF lens are defined as:

$\begin{matrix}{{{LP}\mspace{14mu} {lens}\text{:}}{{{{if}\mspace{14mu} 0} \leq \theta \leq {\theta_{0}\mspace{11mu} ({fovea})}},{r = {r_{\max}f_{lp}\tan \; \theta}},{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{0}} \leq \theta \leq {\theta_{\max}\mspace{14mu} ({periphery})}},}} & (28) \\{r = {r_{\max}{\left\{ {{\log_{a_{lp}}\left( {f_{lp}\tan \; \theta} \right)} + d_{lp}} \right\}.{where}}\mspace{14mu} d_{lp}\mspace{14mu} {is}\mspace{14mu} {denoted}\mspace{14mu} {as}}} & (29) \\{{d_{lp} = {{f_{lp}\tan \; \theta_{0}} - {\log_{a_{lp}}\left( {f_{lp}\tan \; \theta_{0}} \right)}}},{a\mspace{14mu} {focal}\mspace{14mu} {length}\mspace{14mu} f_{lp}\mspace{14mu} {is}\mspace{14mu} {denoted}\mspace{14mu} {as}}} & (30) \\{{f_{lp} = \frac{1}{\left\{ {1 + {\log \left( {\tan \; {\theta_{\max}/\tan}\; \theta_{0}} \right)}} \right\} \tan \; \theta_{0}}},{a\mspace{14mu} {basis}\mspace{14mu} a_{lp}\mspace{14mu} {is}\mspace{14mu} {denoted}\mspace{14mu} {as}}} & (31) \\{{a_{lp} = {\exp \left( \frac{1}{f_{lp}\tan \; \theta_{0}} \right)}},} & (32)\end{matrix}$

such that Equs. (47) and (48) are continuous at θ=θ₀, and theirderivatives are also continuous. Note that the LP lens is equivalentwith the disclosed AdWAF model when θ₁=θ₂=θ_(max), c₀=c₁=1 and c₂=c₃=0.Its fovea and ‘periphery’ correspond to the fovea and para-fovea of thedisclosed AdWAF model, respectively.

$\begin{matrix}{{F\; E\mspace{14mu} {lens}\text{:}}{{r = {\frac{r_{\max}}{\theta_{\max}}\theta \mspace{14mu} \left( {\theta \leq \theta \leq \theta_{\max}} \right)}},}} & (33) \\{{{P\; H\; C\mspace{14mu} {lens}\text{:}}r = {\frac{r_{\max}}{\tan \; \theta_{\max}}\mspace{14mu} \left( {\theta \leq \theta \leq \theta_{\max}} \right)}},{{WAF}\mspace{14mu} {lens}\text{:}}} & (34) \\{r = {{r_{\max}\left( {{a_{0}\theta^{3}} + {a_{1}\theta^{2}} + {a_{2}\theta}} \right)}\mspace{14mu} {\left( {0 \leq \theta \leq \theta_{\max}} \right).}}} & (35)\end{matrix}$

A bold solid line shows the WAF lens. The distribution of its imageheight and M.F. is characterized by the design concept of the WAF lens,that is, acquiring much wider FOV and higher resolution locally in thecentral FOV. Its M.F in the radial direction is much higher than that ofthe PHC lens (a bold broken line) and the FE lens (a fine broken line)in small incident angles, on the other hand, lower than those of the PHCand FE lenses in large incident angles. Comparing with the LP lens (asolid line), one notes that the disclosed AdWAF model (a fine solid linewith circle) can acquire higher M.F. in the central area (0≦h≦h₀), bythe same number of data, because of the AdWAF model having a lower M.F.in the intermediate and peripheral areas. The modification factor c_(i)is used for the M.F. of the AdWAF image in order not to exceed that ofthe WAF lens. Additionally, if r_(max) is 0.93 in the AdWAF model, themodificative M.F. is nearly equivalent to the M.F. of the LP lens in thecentral area, as shown in FIG. 22( b). Therefore, this means that theAdWAF model can reduce more pixels in the peripheral FOV than the LPlens.

The entire FOV is divided into three areas, that is, the central area(0≦θ≦θ₀), the intermediate area (θ₀≦θ≦θ₂) and the peripheral area(θ₂≦θ≦θ_(max)) as shown in FIG. 21. The AdWAF model divides these threeareas further into four areas by a boundary θ₂ between the PP and theSP. The PP and SP parts in the logarithmic area are named para-fovea andnear-periphery, respectively. The para-fovea should preferably be usedfor central vision such as pattern recognition, and the near-peripheryshould preferably be used more for peripheral vision. The central area(fovea) is a planar Cartesian area, because the image height r is linearto the object height h in this area. On the other hand, the peripheralarea (periphery) is a spherical Cartesian area, because the r is linearto the incident angle θ.

B. Image Extraction

FIG. 23 simulates the AdWAF image, by a whole view, under conditions ofr_(max)=64[pixel], θ_(max)=π/3, h_(max)=1, h₀=0.1, h₁=0.4, and h₂=0.6.Each image is extracted from the original image (FIG. 23( a)) having512×512[pixel]. It should be noted that the AdWAF image obviously has ahigher resolution in its central area than the LP lens (FIG. 3). On theother hand, the resolution of its peripheral area is of an order betweenthe WAF lens and the LP lens (FIG. 3).

FIG. 24 shows the AdWAF image, actually extracted from the WAF lensunder the same conditions as of the above simulation. FIG. 24( a), (b)and (c) are the extracted AdWAF images by the whole view, the para-foveaimage, that is, by (17), and the fovea image by (16), respectively. Thefovea image with Cartesian coordinates has only translation-invariantproperty.

C. Examination (i). Relation to K Lens

FIG. 25 shows a comparison of the AdWAF model and Kuniyoshi's lens (Klens) as related to the disclosed AdWAF model, under conditions ofr_(max)=1, θ_(max)=π/3, h_(max)=1, h₀=0.026, h₁=0.21, and h₂=0.6, whichare determined by the distribution of M.F. of an actual K lens. Thevalues of h₁ and h₂, respectively, correspond to incident angles, θ₀(=2.584 [°]) and θ₁ (=20.0 [°]), from boundaries in the K lens' FOV. TheFE lens, the PHC lens, the K lens and the AdWAF model are shown by afine broken line, a bold broken line, a bold solid line, and a finesolid line with circle, respectively. The K lens changes the imageheight logarithmically of θ in h₀≦h≦h₁, and linearly to θ inh₁≦h≦h_(max). In the case when the first boundary h₀ (=θ₀) is equal withthe second boundary h₁ (=θ₁) and the third boundary h₂ (=θ₂) is equalwith the second boundary of the K lens model they are consistentalthough Kuniyoshi assumes that two focal lengths, f₁ and f₂, to the PPand SP surfaces have the same value. It may be noted that this conditionof the boundaries gives the disclosed AdWAF model a higher M.F. in itscentral area than the K lens, by the same number of data. On the otherhand, if it has the same M.F. in its central area as the K lens, ther_(max) is 0.85. Therefore, the disclosed AdWAF model in this case canreduce the number of data by about 28 percent.

(ii). Simulation Results

FIG. 26 shows test images with each different scale (α=0.75, 1 and 1.5)as related to the disclosed AdWAF model, extracted from FIG. 26( a)under conditions of θ_(max)=π/3, h_(max)=1, h₀=0.026, h₁=0.21, andh₂=0.6.

Both of the disclosed AdWAF model and the Kuniyoshi's model can acquireLP image with rotation and scale-invariant property. FIGS. 27 and 6 showa comparison of the LP lens (left) and the AdWAF model (right), and acomparison of the LP lens (left) and the K lens (right), respectively,by their polar coordinate images extracted from test images in FIG. 26(a), (b) and (c). Both comparisons are simulated under the sameconditions of θ_(max)=π/3, h_(max)=1, h₀=0.026, h₁=0.21, and h₂=0.6. Abright part in each image shows the logarithmic area composed ofpara-fovea and near-periphery. On the other hand, upper and lower darkparts, where brightness of each pixel is reduced by half, show theplanar Cartesian area (fovea) and the spherical Cartesian area(periphery). One notes that the disclosed AdWAF model acquires the widerfovea with translation-invariance than the LP lens, in spite of beingthe periphery (the lower dark part), because the near-periphery reducesthe image size of its corresponding part. In this condition, thedisclosed AdWAF model acquires the wider fovea and a wider logarithmicarea than that acquired by the K lens.

FIG. 28 shows a length on the image plane to indicate accuracy ofscale-invariance in terms of the object height, h, as related to the Klens (AdWAF) and the disclosed AdWAF model's image. This length is adifference between an image height corresponding to each h and anotherimage height corresponding to 95 percent of the h, where r_(max)=1. Abroken line, a bold solid line and a fine solid line with circle showthe LP lens, the K lens and the AdWAF model, respectively.Scale-invariance means that a gradient of each line is continuouslyzero. One notes that all are not scale-invariant in the fovea (0≦h≦h₀),consequently. In para-fovea, the K-lens is not scale-invariant, exactly.A fine solid line shows the simulated K lens drawn by the disclosedAdWAF model under a different condition of θ_(max)=π/3, h_(max)=1,h₀=0.026, h₁=0.026, and h₂=0.6. This line changes more smoothly thanthat of the original K lens, although para-fovea is not scale-invariant.Thus, since the definition of the disclosed AdWAF model is moreaccurate, the disclosed AdWAF model can describe other WAF visionsensors more flexibly.

The foregoing demonstrates that (a) the disclosed AdWAF model canacquire a higher M.F. in the fovea than the LP lens model, (b) thedisclosed AdWAF model can acquire a more accurate scale-invariant areain the para-fovea and can acquire a wider translation-invariant area inthe fovea, and (c) the disclosed AdWAF model can describe another WAFlens more flexibly, because the AdWAF model is defined more accurately.

III. Another Embodiment of the AdWAF Model A. Modeling

In order to make a better all-purpose use of the WAF image, ageometrical model, namely, another embodiment (AdWAF) model isdisclosed. FIG. 29 shows another embodiment of a camera model thatcombines planar projection and spherical projection. The former is aperspective projection, that is, linear to tangent of incident angle θto the lens optical center, and the latter is linear to the θ. Theprojection height, p, of this camera model is represented as below:

if 0≦θ≦θ₁,

p=f₁ tan θ,  (36)

else if θ₁≦θ≦θ_(max),

p=f ₂(θ−θ₁)+f ₁ tan θ₁,  (37)

where f₁ and f₂ are focal lengths to the projection plane and thespherical projection surface, respectively.

The disclosed AdWAF model is denoted by the following equations,combining both planar projection by (36) and spherical projection by(37) with both linear coordinates and logarithmic coordinates.

if 0≦θ≦θ₀,

r=r_(max)c₀f₁ tan θ,  (38)

else if θ₀≦θ≦θ₁,

r=r _(max) {c ₁ log_(a)(f ₁ tan θ)+d ₁},  (39)

else if θ₁≦θ≦θ₂,

r=r _(max) {c ₂ log_(b)(f ₂θ)+d ₂},  (40)

else if θ₂≦θ≦θ_(max),

r=r _(max)(c ₃ f ₂ θ+d ₃),  (41)

where r is image height versus the θ, r_(max) is the maximum imageheight when θ=θ_(max), c_(i) (i=0, 1, 2, 3) is a scale modificationfactor for adjusting image height partly in each section of θ and d_(i)(i=0, 1, 2, 3) is

d ₁ =c ₀ f ₁ tan θ₀ −c ₁ log_(a)(f ₁ tan θ₀),  (42)

d ₂ =c ₁ log_(a)(f ₁ tan θ₁)−c ₂ log_(b)(f ₂θ₁)+d ₁,  (43)

d ₃ =c ₂ log_(b)(f ₂θ₂)−c ₃ f ₂θ₂ +d ₂.  (44)

Because Equs. (38) to (41) are continuous at each boundary, if thesederivatives are also continuous when c₀=c₁=c₂=c₃=1,

$\begin{matrix}{{f_{1} = {\frac{1}{\tan \; \theta_{0}}/\left\{ {1 + {\log \; \frac{\tan \; \theta_{1}}{\tan \; \theta_{0}}} + {\frac{\theta_{1}}{\cos \; \theta_{1}\sin \; \theta_{1}}\left( {\frac{\theta_{\max} - \theta_{2}}{\theta_{2}} + {\log \; \frac{\theta_{2}}{\theta_{1}}}} \right)}} \right\}}},} & (45) \\{{f_{2} = {\frac{f_{1}\tan \; \theta_{0}}{\cos \; \theta_{1}\sin \; \theta_{1}} \cdot \frac{\theta_{1}}{\theta_{2}}}},} & (46) \\{{a = {\exp \left( \frac{1}{f_{1}\tan \; \theta_{0}} \right)}},} & (47) \\{b = {{\exp \left( \frac{1}{f_{2}\theta_{2}} \right)}.}} & (48)\end{matrix}$

The disclosed AdWAF model divides the field of view into four areas,that is, fovea (0≦θ≦θ₀), para-fovea (θ₀≦θ≦θ₁), near-periphery (θ₁≦θ≦θ₂),and periphery (θ₂≦θ≦θ_(max)). The fovea is planar and its image heightis linear to the object height h. On the other hand, the periphery isspherical and its image height is linear to the incident angle θ. FIG.30 simulates an image by the disclosed AdWAF model and a Cartesianlinear-coordinate image by pinhole camera (PHC) lens model, undercondition that the boundaries of FOV, θ₀, θ₁, and θ₂, are 9.826 [°],19.107 [°], and 34.715 [°], respectively. The intensity is changed inorder to see each boundary easily.

FIG. 31 shows the image height r, M.F. dr/dh and r/h in the radial andtangential directions of the disclosed AdWAF model for an AdWAF imageplot, versus the object height h. The h_(max) and r_(max) are normalizedto 1 (when θ_(max)=π/3) in order to compare other types of lens, thatis, a log-polar (LP) lens, a fish eye (FE) lens, the PHC lens and theWAF lens. In this simulation, the boundaries of FOV, that is, h₀, h₁ andh₂, are 0.1 (θ₀=9.826 [°]), 0.4 (θ₁=19.107 [°]), and 0.6 (θ₂=34.715[°]), respectively.

A bold solid line shows the actual WAF lens. The distribution of itsimage height and M.F. is characterized by the design concept of the WAFlens, that is, acquiring wide FOV and high resolution locally in thecentral FOV. Its M.F. in the radial direction is much higher than thatof the PHC lens (a bold broken line) and the FE lens (a fine brokenline) in small incident angles, on the other hand, lower in largeincident angles. FIG. 31 shows that the disclosed AdWAF model (a finesolid line with circle) can acquire a higher M.F. in the fovea 0≦h≦h₀(that is, 0≦θ≦θ₀) than the LP lens (a solid line), in the case of thesame FOV. The scale modification factor c_(i) is applicable foradjusting the image height of the disclosed AdWAF image in order to makeits M.F. in the fovea to be equal to that of the LP lens. Ifc₀=c₁=c₂=c₃=0.93, the modified M.F. is almost equal to that of the LPlens in the fovea in the case of FIG. 31( b). Stated differently, thismeans that the disclosed AdWAF model may reduce the number of pixels byabout 13.5 percent in the whole of image comparing to that by the LPlens.

B. Implementation

FIG. 32 simulates an image by the disclosed AdWAF model (AdWAF image),by the whole view, under conditions of r_(max)=64[pixel], θ_(max)=π/3,θ₀=9.826 [°], θ₁=19.107 [°] and θ₂=34.715 [°]. Each image is simulatedfrom a target image of 512×512 [pixels] (FIG. 32( a)). A comparison withexisting lenses has been done elsewhere, such as FIG. 3. The AdWAF image(FIG. 32( f)) has a higher resolution in its central area than the LPlens image (FIG. 32( c)). On the other hand, the resolution of itsperipheral area is between those of the WAF lens (FIG. 32( b)) and theLP lens. It should be remarked that all of these simulated images can berepresented using the AdWAF model.

FIG. 33 shows an AdWAF image, actually extracted from a WAF lens underthe same conditions as those of the above simulation. FIG. 33( a), (b),(c) and (d) are an actual input image by the WAF lens, the extractedAdWAF image by the whole view, the para-fovea image, that is, alog-polar image (with planar logarithmic coordinates) by (39), and thefovea image (with planar linear coordinates) by (38), respectively. Inaddition to wide FOV, the rotation- and scale-invariant property of thepara-fovea image, and translation-invariant property of the fovea imageare suitable for an all-purpose use.

C. Examination (i). Representing Other Foveation Models

Some foveation models used for the existing log-polar chip and visionsystem are represented by the disclosed AdWAF model when θ₁=θ₂=θ_(max)and c₂=c₃=0. The FOV of such models is divided into fovea and“periphery” similarly to the LP lens. The para-fovea of the disclosedAdWAF model denotes a log-polar grid in “periphery,” assuming that thePHC lens is used. The log-polar grid is composed of rings and rays forposition of receptive fields (RF's) (FIG. 34A). On the other hand, thefovea has a uniform size of the RF's. In addition, this size is equal tothat of the first ring in ‘periphery’, in order to avoid discontinuityat the fovea ‘periphery’ boundary. A radius of each ring is calculatedas the normalized object height h using the disclosed AdWAF model asfollows:

if 0≦θ≦θ₀ (fovea),

$\begin{matrix}{{h = \frac{r}{r_{\max}c_{0}f_{1}\tan \; \theta_{\max}}},{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{0}} \leq \theta \leq {\theta_{\max}\mspace{14mu} \left( {``{periphery}"} \right)}},} & (49) \\{{h = {\frac{\tan \; \theta_{0}}{\tan \; \theta_{\max}}a^{\frac{r - r_{0}}{r_{\max}c_{1}}}}},} & (50)\end{matrix}$

where r₀ is a radius of the fovea\“periphery” boundary, that is,

r₀=r_(max)c₀f₁ tan θ₀.  (51)

With respect to the log-polar sensor, one of the differences between thelens and the solid-state chip is the number, N, of the RF's along thering. Here, the case of the lens assumes that each photosensitiveelement is equivalent with the RF. The number N of the LP lensincreases, as the h gets larger (in population to the r), while thelog-polar chip has a constant number N₀ in “periphery.” FIG. 34A(a)compares the N of both cases versus the h, when N₀=128, θ₀=9.826 [°],θ_(max)=60.0 [°] and c₀=c₁=1, in addition to that, when the number ofthe RF's is equal in the fovea.

Comparing Sandini's model (FIG. 34A(b)) and Bolduc's model (FIG.34A(c)), the size of RF's changes differently between these two modelsin a hatched area of the FOV (that is, “periphery”), although the N₀, θ₀and θ_(max) are common. This means a ring number is not equal to theimage height r, necessarily, because each RF could be composed ofmultiple photosensitive elements. The disclosed AdWAF model representsdifferent arrangement of the log-polar grid by the scale modificationfactors c₀ and c₁. The c₀ modifies the r₀ (that is, modifies the numberof the RF's in the fovea). The c₁ adjusts logarithmic change of a radiusof the ring. In this case, the r in Equs. (49) and (50) can be regardedas the ring number. Thus, the c₀ and c₁ fit both models into thedisclosed AdWAF model even with the same N₀, θ₀ and θ_(max).

The Kuniyoshi lens (K lens) model has a planar linear part in 0≦θ≦θ₀, aspherical logarithmic part in θ₀≦θ≦θ₁ and a spherical linear partθ₁≦θ≦θ_(max), but it does not have the planar logarithmic part(para-fovea by Equ. (39)). Thus, the disclosed AdWAF model representsthe K lens model, in condition of f_(k1)=f₁ and f_(k2)=f₂, as follows:

K Lens Model:

$\begin{matrix}{{{{if}\mspace{14mu} 0} \leq \theta \leq \theta_{0}},{r = {r_{\max}f_{k\; 1}\tan \; \theta}},{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{0}} \leq \theta \leq \theta_{1}},} & (52) \\{{r = {r_{\max}\left\{ {{\log_{b_{k}}\left( {f_{k\; 2}\theta} \right)} - p} \right\}}},{{{else}\mspace{14mu} {if}\mspace{14mu} \theta_{1}} \leq \theta \leq \theta_{\max}},} & (53) \\{{r = {r_{\max}\left( {{f_{k\; 2}\theta} + q} \right)}},{where}} & (54) \\{{b_{k} = {\exp \left\{ {\frac{1}{1 - {f_{k\; 1}\tan \; \theta_{0}}}\left( {\frac{\theta_{\max} - \theta_{1}}{\theta_{1}} + {\log \; \frac{\theta_{1}}{\theta_{0}}}} \right)} \right\}}},} & (55) \\{{p = {{\log_{b_{k}}\left( {f_{k\; 2}\tan \; \theta_{0}} \right)} - {f_{k\; 1}\tan \; \theta_{0}}}},} & (56) \\{q = {{{- f_{k\; 2}}\theta_{1}} + {\log_{b_{k}}\frac{\theta_{1}}{\theta_{0}}} + {f_{k\; 1}\tan \; {\theta_{0}.}}}} & (57)\end{matrix}$

FIG. 34B shows a comparison of the disclosed AdWAF model (a fine solidline with circle) and the K lens (a bold solid line), in condition ofr_(max)=1, θ_(max)=60.0 [°], θ₀=2.584 [°], θ₁=20.0 [°] and θ₂=34.715[°], related to the AdWAF Image plot (these values are determined fromspecification of the actual K lens). The FE lens (a fine broken line)and the PHC lens (a bold broken line) are also compared with them. Thiscondition of boundaries indicates the AdWAF image gives a higher M.F. in0≦θ≦θ₀ than the K lens, by the same FOV. On the other hand, when it hasthe same M.F. in the fovea (0≦θ≦θ₀) as the K lens, the r_(max) is 0.85.This means that the AdWAF image reduces the number of pixels by about 28percent.

(ii). Comparison of AdWAF Image, LP Lens and K Lens

FIG. 35( i) compares the AdWAF image, the LP lens image and the K lensimage, simulated under a condition of θ_(max)=60.0 [°], θ₀=2.584 [°],θ₁=20.0 [°] and θ₂=34.715 [°], by changing intensity. The FOV is dividedinto four areas, that is, fovea (the central dark part), para-fovea (thebrightest part), near-periphery (the second brightest part) andperiphery (outer dark part). It is noted that the LP lens image has onlyfovea and para-fovea, and that the K lens image does not havepara-fovea. The AdWAF image shows a higher M.F. than the K lens image inthe fovea.

FIG. 35( ii) shows three target images with different scales (α=0.75, 1and 1.5). FIG. 35( iii) compares polar images of the AdWAF image (left),the LP lens image (middle) and the K lens image (right), in each scale(under the same condition as in FIG. 35( i)). It is noted that the polarimage of the para-fovea has a translation-invariant property in theradial and tangential directions, that is, this part has rotation- andscale-invariant (RS-invariant) property, when the r is linear to theobject height h. On the other hand, the near-periphery is RS-invariantwhen the r is linear to the incident angle θ. These two types ofRS-invariant property give the AdWAF image the following advantages:

1) The para-fovea is suitable for matching images with different scale.2) The near-periphery reduces influence by rotation of the object plane(that is, this causes skew).

That is, these advantages indicate that the disclosed AdWAF modelmatches different-scaled patterns accurately in wider incident angles,and that it is more robust to the camera rotation.

In order to estimate scale-invariant property, a length Δr on the imageplane is calculated from the object height h and its 95-percent heighth′ (as in FIG. 36( a)). FIG. 36( b) shows the Δr versus the h, incondition of r_(max)=1, θ_(max)=60.0 [°], θ₀=2.584 [°], θ₁=20.0 [°] andθ₂=34.715 [°]. A fine solid line with circle, a broken line and a boldsolid line show the disclosed AdWAF model, the LP lens and the K lens,respectively. If a gradient of these lines is constantly zero, acorresponding part (that is, a planar logarithmic part) isscale-invariant to the planar projection of the object plane. Thus, theimages by the LP lens and the disclosed AdWAF model are scale-invariantin θ₀≦θ≦θ_(max) and in θ₀≦θ≦θ₁, respectively. The K lens image is notscale-invariant exactly in this view, that is, the Δr has 7.6-percenterror at most in θ₀≦θ≦0 ₁ (a spherical logarithmic part of the K lens).

In addition, the boundary θ₁=20.0 [°] (h₁=0.21) of the actual K lenscurve is not suitable for image matching, especially in neighborhood ofthe boundary θ₁. FIG. 36( b) compares the actual curve and the simulatedK lens curve (a fine solid line without circle), based on the disclosedAdWAF model, in condition of θ_(max)=60.0 [°], θ₀=2.584 [°] andθ₁=34.715 [°].

Rotation of the object plane causes skew. In order to test itsinfluence, cosine of a skew coefficient a_(c) is defined. FIG. 37( a)shows a sketch from rotation ψ to the α_(c).

$\begin{matrix}{{\cos \left( \alpha_{c} \right)} = {\frac{\sin^{2}\psi}{1 + {\cos^{2}\psi}}.}} & (58)\end{matrix}$

If the ψ is zero, the optical axis is perpendicular to the object plane,thus, α_(c)=π/2 (cos(α_(c))=0). FIG. 37( b) compares the disclosed AdWAFmodel and the LP lens in terms of root mean square error (RMSE),calculated from a part (θ₀≦θ≦θ₂) of each polar image, using the case ofψ=0 as a model. The value of cos(α_(c)) from 0 to 15.3×10⁻³ correspondsto the ψ from 0 to 10 [°]. The disclosed AdWAF model has a smaller RMSEthan the LP lens. This means it is more robust to the skew than the LPlens, because the spherical logarithmic part of θ₁≦θ≦θ₂ gives us lessimage deformation.

FIG. 38 compares the disclosed AdWAF model and the LP lens model bytheir polar images when ψ=0, 5 and 10 [°]. The polar image deformstoward the directions that white arrows show, as the ψ gets larger. FIG.38 also shows the deformation by the disclosed AdWAF model is smallerthan that by the LP lens model.

FIG. 39 illustrates a flowchart of a method 3900 of the presentdisclosure. The method 3900 of modeling an image may include providingan optical element (block 3902), assigning a field of view of theoptical element (block 3904), sectoring the field of view in a pluralityof areas (block 3906), and processing an image in accordance with theplurality of sectored areas (block 3908). In the method 3900, theprocessing the image further may include processing at least oneprojection selected from: a planar projection, a planar logarithmicprojection, a spherical logarithmic projection, a spherical projection,and a log-polar projection. In the method 3900, the processing the imagemay further include processing at least one coordinate selected from: aCartesian coordinate, a spherical coordinate, and a logarithmiccoordinate.

In the method 3900, the processing the image may further includerendering the image to have a feature selected from the following:rotation-invariant, scale-invariant, and translation-invariant.

In the method 3900, the sectoring the field of view in the plurality ofareas may further include sectoring the field of view at least in anarea selected from: a foveal area, a para-foveal area, a near-peripheralarea, and a peripheral area.

In the method 3900, the providing the optical element may furtherinclude providing a lens. The providing the lens may further includeproviding one of a log-polar lens and a mirror. In the method 3900, theproviding the log-polar lens may further include applying a planarprojection in a central area of a field of view. The providing thelog-polar lens may further include applying a spherical projection in aperipheral area of a field of view.

In the method 3900, the providing the lens may include providing afish-eye lens, a pin hole camera lens, and a wide-angle foveated lens.Further, the method may further include sensing an object in the fieldof view, the sensing being performed in accordance with at least one ofa plurality of sectors of the field of view. In the method 3900, theprocessing the image may further include storing the image, detecting anedge in a peripheral area, transforming into a perspective coordinate,detecting a straight line by performing Hough transform, determining aboundary line, locating a gaze direction, and controlling a camera.

The foregoing method 3900 or elements of the method 3900 could also bestored on a computer-readable medium having computer-executableinstructions to implement the method 3900 or the elements of the method3900.

FIG. 40 illustrates a block diagram of an exemplary embodiment of anapparatus 4000 of the present disclosure. The apparatus 4000 may includean optical element 4002 having a field of view, an image sectoringelement 4004 coupled to the optical element 4002, the image sectoringelement 4004 configured to sector the field of view in a plurality ofareas, and an image processor 4006 coupled to the image sectoringelement 4004, the image processor 4006 configured to process an image inaccordance with the plurality of areas.

In the apparatus 4000, the image processor 4006 may be furtherconfigured to process at least one projection selected from: a planarprojection, a planar logarithmic projection, a spherical logarithmicprojection, a spherical projection, and a log-polar projection. Theimage processor 4006 may be further configured to process at least onecoordinate selected from: a Cartesian coordinate, a sphericalcoordinate, and a logarithmic coordinate.

In the apparatus 4000, the image processor 4006 may also be furtherconfigured to render the image to have a feature selected from: arotation-invariant feature, a scale-invariant feature, and atranslation-invariant feature.

The image sectoring element 4004 may be further configured to sector thefield of view at least in an area selected from: a foveal area, apara-foveal area, a near-peripheral area, and a peripheral area.

The optical element 4002 may be configured to apply at least one of aplanar projection in a central area of the field of view and a sphericalprojection in a peripheral area of the field of view. The opticalelement 4002 may include one of a fish-eye lens, a pin hole camera lens,a mirror, and a wide-angle foveated lens.

The apparatus 4000 may further include a sensor coupled to the imageprocessor 4006, the sensor configured to sense at least one of aplurality of sectors of a field of view. In the apparatus 4000, theimage processor 4006 may be further configured to store the image,detect an edge in a peripheral area, transform into a perspectivecoordinate, detect a straight line by performing Hough transform,determine a boundary line, locate a gaze direction, and control acamera.

As used in this specification and appended claims, the singular forms“a,” “an,” and “the” include plural referents unless the specificationclearly indicates otherwise. The term “plurality” includes two or morereferents unless the specification clearly indicates otherwise. Further,unless described otherwise, all technical and scientific terms usedherein have meanings commonly understood by a person having ordinaryskill in the art to which the disclosure pertains.

As a person having ordinary skill in the art would appreciate, theelements or blocks of the methods described above could take place atthe same time or in an order different from the described order.

It should be emphasized that the above-described embodiments are merelysome possible examples of implementation, set forth for a clearunderstanding of the principles of the disclosure. Many variations andmodifications may be made to the above-described embodiments of theinvention without departing substantially from the principles of theinvention. All such modifications and variations are intended to beincluded herein within the scope of this disclosure and the presentinvention and protected by the following claims.

1. A method of modeling an image, the method comprising: providing anoptical element; assigning a field of view of the optical element;sectoring the field of view in a plurality of areas; and processing animage in accordance with the plurality of sectored areas.
 2. The methodof claim 1, wherein the processing the image further includes processingat least one projection selected from the group consisting of: a planarprojection, a planar logarithmic projection, a spherical logarithmicprojection, a spherical projection, and a log-polar projection.
 3. Themethod of claim 1, wherein the processing the image further includesprocessing at least one coordinate selected from the group consistingof: a Cartesian coordinate, a spherical coordinate, and a logarithmiccoordinate.
 4. The method of claim 1, wherein the processing the imagefurther includes rendering the image to have a feature selected from thegroup consisting of: rotation-invariant, scale-invariant, andtranslation-invariant.
 5. The method of claim 1, wherein the sectoringthe field of view in the plurality of areas further includes sectoringthe field of view at least in an area selected from the group consistingof: a foveal area, a para-foveal area, a near-peripheral area, and aperipheral area.
 6. The method of claim 1, wherein the providing theoptical element includes providing a lens.
 7. The method of claim 6,wherein the providing the lens includes providing one of a log-polarlens and a mirror.
 8. The method of claim 7, wherein the providing thelog-polar lens further includes applying a planar projection in acentral area of a field of view.
 9. The method of claim 7, wherein theproviding the log-polar lens further includes applying a sphericalprojection in a peripheral area of a field of view.
 10. The method ofclaim 6, wherein the providing the lens includes providing a fish-eyelens.
 11. The method of claim 6, wherein the providing the lens includesproviding a pin hole camera lens.
 12. The method of claim 6, wherein theproviding the lens includes providing a wide-angle foveated lens. 13.The method of claim 1, wherein the method further includes sensing anobject in the field of view, the sensing being performed in accordancewith at least one of a plurality of sectors of the field of view. 14.The method of claim 1, wherein the processing the image furtherincludes: storing the image; detecting an edge in a peripheral area;transforming into a perspective coordinate; detecting a straight line byperforming Hough transform; determining a boundary line; locating a gazedirection; and controlling a camera.
 15. An apparatus for imageprocessing, comprising: an optical element having a field of view; animage sectoring element coupled to the optical element, the imagesectoring element configured to sector the field of view in a pluralityof areas; and an image processor coupled to the image sectoring element,the image processor configured to process an image in accordance withthe plurality of areas.
 16. The apparatus of claim 15, wherein the imageprocessor is further configured to process at least one projectionselected from the group consisting of: a planar projection, and a planarlogarithmic projection, a spherical logarithmic projection, a sphericalprojection, and a log-polar projection.
 17. The apparatus of claim 15,wherein the image processor is further configured to process at leastone coordinate selected from the group consisting of: a Cartesiancoordinate, a spherical coordinate, and a logarithmic coordinate. 18.The apparatus of claim 15, wherein the image processor is furtherconfigured to render the image to have a feature selected from the groupconsisting of: rotation-invariant, scale-invariant, andtranslation-invariant.
 19. The apparatus of claim 15, wherein the imagesectoring element is further configured to sector the field of view atleast in an area selected from the group consisting of: a foveal area, apara-foveal area, a near-peripheral area, and a peripheral area.
 20. Theapparatus of claim 15, wherein the optical element is configured toapply at least one of: a planar projection in a central area of thefield of view and a spherical projection in a peripheral area of thefield of view.
 21. The apparatus of claim 15, wherein the opticalelement includes one of a fish-eye lens, a pin hole camera lens, amirror, and a wide-angle foveated lens.
 22. The apparatus of claim 15,wherein the apparatus further includes a sensor coupled to the imageprocessor, the sensor configured to sense at least one of a plurality ofsectors of a field of view.
 23. The apparatus of claim 15, wherein theimage processor is further configured to store the image; detect an edgein a peripheral area; transform into a perspective coordinate; detect astraight line by performing Hough transform; determine a boundary line;locate a gaze direction; and control a camera.
 24. A computer-readablemedium having computer-executable instructions for: providing an opticalelement; assigning a field of view of the optical element; sectoring thefield of view in a plurality of areas; and processing an image inaccordance with the plurality of sectored areas.